Avoiding the Look-Ahead Pathology of Decision Tree Learning

Most decision-tree induction algorithms are using a local greedy strategy, where a leaf is always split on the best attribute according to a given attribute selection criterion. A more accurate model could possibly be found by looking ahead for alternative subtrees. However, some researchers argue that the look-ahead should not be used due to a negative effect (called ―decision tree pathology‖) on the decision tree accuracy. This paper presents a new look-ahead heuristics for decision-tree induction. The proposed method is called LA-J48 (―Look-Ahead J48‖) as it is based on J48, the Weka implementation of the popular C4.5 algorithm. At each tree node, the LA-J48 algorithm applies the look-ahead procedure of bounded depth only to attributes that are not statistically distinguishable from the best attribute chosen by the greedy approach of C4.5. A bootstrap process is used for estimating the standard deviation of splitting criteria with unknown probability distribution. Based on a separate validation set, the attribute producing the most accurate subtree is chosen for the next step of the algorithm. In experiments on 20 benchmark datasets, the proposed look-ahead method outperforms the greedy J48 algorithm with the Gain Ratio and the Gini Index splitting criteria, thus avoiding the look-ahead pathology of decision tree induction.